Despite its theoretical appeal, this viewpoint lacks a crucial ingredient of deep learning in finite DNNs, laying at the heart of their success --feature learning. Here we consider DNNs trained with noisy gradient descent on a large training set and derive a self-consistent Gaussian Process theory accounting forstrongfinite-DNN and feature learning effects.

Machine learning has seen many successful achievements in recent years.

Self-Consistent Equation-guided Neural Networks for Censored Time-to-Event Data Sehwan Kim 1, Rui Wang 1,2, and Wenbin Lu 3 1 Department of Population Medicine, Harvard Pilgrim Health Care Institute and Harvard Medical School, Boston, MA 2 Department of Biostatistics, Harvard School of Public Health, Boston, MA 3 Department of Statistics, North Carolina State University, Raleigh, NC March 13, 2025 Abstract In survival analysis, estimating the conditional survival function given predictors is often of interest. There is a growing trend in the development of deep learning methods for analyzing censored time-to-event data, especially when dealing with high-dimensional predictors that are complexly interrelated. Many existing deep learning approaches for estimating the conditional survival functions extend the Cox regression models by replacing the linear function of predictor effects by a shallow feed-forward neural network while maintaining the proportional hazards assumption. Their implementation can be computationally intensive due to the use of the full dataset at each iteration because the use of batch data may distort the at-risk set of the partial likelihood function. To overcome these limitations, we propose a novel deep learning approach to non-parametric estimation of the conditional survival functions using the generative adversarial networks leveraging self-consistent equations. The proposed method is model-free and does not require any parametric assumptions on the structure of the conditional survival function. We establish the convergence rate of our proposed estimator of the conditional survival function. In addition, we evaluate the performance of the proposed method through simulation studies and demonstrate its application on a real-world dataset. 1 Introduction Censored time-to-event data are widely encountered in various fields where understanding the timing of events, such as failure rates or disease progression, is critical, but the exact event times Correspondence author: Wenbin Lu, email: wlu4@ncsu.edu 1 arXiv:2503.09097v1 For example, estimating survival probability based on covariate information is essential for risk prediction, which plays a key role in developing and evaluating personalized medicine. The Kaplan-Meier (KM) estimator (Kaplan and Meier, 1958), Cox proportional hazards model (Cox, 1972), and random survival forests (Ishwaran et al., 2008) are commonly-used methods for estimating survival functions. The KM estimator is a non-parametric method suitable for population-level analyses. However, its utility is limited when the objective is to estimate conditional survival probabilities at the individual level. The Cox proportional hazards model offers a semi-parametric approach for estimating conditional survival functions, accommodating the incorporation of covariates.

Graph neural networks (GNNs) are designed to process data associated with graphs. They are finding an increasing range of applications; however, as with other modern machine learning techniques, their theoretical understanding is limited. GNNs can encounter difficulties in gathering information from nodes that are far apart by iterated aggregation steps. This situation is partly caused by so-called oversmoothing; and overcoming it is one of the practically motivated challenges. We consider the situation where information is aggregated by multiple steps of convolution, leading to graph convolutional networks (GCNs). We analyze the generalization performance of a basic GCN, trained for node classification on data generated by the contextual stochastic block model. We predict its asymptotic performance by deriving the free energy of the problem, using the replica method, in the high-dimensional limit. Calling depth the number of convolutional steps, we show the importance of going to large depth to approach the Bayes-optimality. We detail how the architecture of the GCN has to scale with the depth to avoid oversmoothing. The resulting large depth limit can be close to the Bayes-optimality and leads to a continuous GCN. Technically, we tackle this continuous limit via an approach that resembles dynamical mean-field theory (DMFT) with constraints at the initial and final times. An expansion around large regularization allows us to solve the corresponding equations for the performance of the deep GCN. This promising tool may contribute to the analysis of further deep neural networks.

A key problem in deep learning and computational neuroscience is relating the geometrical properties of neural representations to task performance. Here, we consider this problem for continuous decoding tasks where neural variability may affect task precision. Using methods from statistical mechanics, we study the average-case learning curves for $\varepsilon$-insensitive Support Vector Regression ($\varepsilon$-SVR) and discuss its capacity as a measure of linear decodability. Our analysis reveals a phase transition in the training error at a critical load, capturing the interplay between the tolerance parameter $\varepsilon$ and neural variability. We uncover a double-descent phenomenon in the generalization error, showing that $\varepsilon$ acts as a regularizer, both suppressing and shifting these peaks. Theoretical predictions are validated both on toy models and deep neural networks, extending the theory of Support Vector Machines to continuous tasks with inherent neural variability.

A sharp asymptotics of the under-bagging (UB) method, which is a popular ensemble learning method for training classifiers from an imbalanced data, is derived and used to compare with several other standard methods for learning from imbalanced data, in the scenario where a linear classifier is trained from a binary mixture data. The methods compared include the under-sampling (US) method, which trains a model using a single realization of the subsampled dataset, and the simple weighting (SW) method, which trains a model with a weighted loss on the entire data. It is shown that the performance of UB is improved by increasing the size of the majority class, even if the class imbalance can be large, especially when the size of the minority class is small. This is in contrast to US, whose performance does not change as the size of the majority class increases, and SW, whose performance decreases as the imbalance increases. These results are different from the case of the naive bagging in training generalized linear models without considering the structure of class imbalance, indicating the intrinsic difference between the ensembling and the direct regularization on the parameters.